The invention generally relates to hand-held communication and computing devices, such as cellular-phone handsets, smart-phones, Web-phones, and PDAs. More specifically, the invention relates to systems and methods for coordinating images displayed on hand-held devices that have two or more displays, especially those that include a direct-view display for viewing relatively low-resolution text or images (like the displays built into most cellular phones today), and a microdisplay that uses magnifying optics to display images that, when the microdisplay is held near a user's, eye can appear larger and higher resolution than the images shown on the direct-view display.
Using recently developed high-resolution microdisplay module technology, device makers can embed microdisplays in devices, or attach microdisplays including wearable displays to devices, and allow users to view full-size web-pages on pocket-size devices—as well as being able to view large video and still images, blueprints, office documents, and other high-resolution content.
Device makers have experimented with embedding microdisplays in devices and attaching microdisplays to devices (including attaching wearable near-to-eye displays). However, prior to this invention, it has been awkward and difficult to navigate and interact with full-size Web pages and other documents and applications displayed on the microdisplays of microdisplay-enhanced hand-held devices. Devices with embedded microdisplays must be held near-to-eye to see the content on the microdisplays. While users do not mind briefly bringing a device near-to-eye to view content (after all, that is what we do with cameras), users do not want to hold a device near to eye for longer periods of time. If a user is spending five minutes, ten minutes, or more reading or interacting with a set of Web pages or other documents, and if they have to hold a device near-to-eye continuously during that time, they will grow uncomfortable with the device. Furthermore, typing text on a hand-held device (already a relative awkward activity) is exceedingly awkward when the device has to be held near-to-eye to see the text as you type it. Devices with attached wearable displays, such as the head-mounted monocular displays that Xybernaut Corp. uses in their wearable computer systems, allow users to comfortably keep microdisplays near-to-eye for longer periods of time (relative to hand-held devices with embedded displays); but users still find it awkward to type text into documents viewed on wearable displays. Typing text into a Web page or other document displayed by a device on a microdisplay typically involves either using a cursor control to “type” on a virtual keyboard image shown on the microdisplay (e.g. moving the cursor over each character displayed on the virtual keyboard and clicking), or using a separate physical device (such as a keypad, a keyboard, a tablet or writing pad, or a virtual keyboard). Typing using a virtual keyboard displayed on the microdisplay is tedious and slow (particularly when holding a device near-to-eye) because it requires the user to carefully position the cursor over each little character on the virtual keyboard to type each character. Further, using a mouse on a regular computer to type text on a virtual keyboard is awkward, and it is even more awkward when using a near-to-eye microdisplay. Even when a user can type text on a physical keypad while looking at a document on a wearable display, if user's haven't learned to touch-type well (i.e. type without looking at the keypad), then users can experience an uncomfortable mental disconnect as they frequently shift their eye (or eyes) between the keypad and the image displayed on the wearable display's near-to-eye microdisplays.
However, the potential benefits of using near-to-eye microdisplays with hand-held devices are tremendous: They can allow users to view full-size web pages and other high-resolution content on pocket-size devices. So solving the problems outlined above would enable the development of compelling new types of hand-held devices. This patent describes inventions that help solve these problems.
“A device with which the invention can be implemented is disclosed and claimed in U.S. published Application No. 20020158812, filed on even date herewith, assigned to the common assignee, and incorporated herein by reference.”
In devices using the present invention, at least one display is a direct-view display, like those on most cell phones or PDAs in use in 2001, for viewing at normal reading distance of approximately 12 to 24 inches (which this patent will refer to as “arms'-length” viewing). And at least one of the displays is a microdisplay, a tiny display with magnifying optical elements, for viewing larger, higher-resolution images when the microdisplay is positioned close to the eye. The microdisplays can be embedded in devices, or they can be attached to the devices, or the microdisplays can be in wearable displays attached to the devices (wirelessly or through a cable). Many digital video cameras use microdisplays for their view-finders: When a user brings the view-finder to their eye when the camera is playing back video, the user can watch a larger image of the video-clip than could be seen on small direct-view display. In recent years, high resolution microdisplays have been developed that can present images large enough to display most Web pages. For example, Inviso Inc.'s 3430 Optiscape II microdisplay displays an 800×600 pixel color image that appears to users to be as large as a 19-inch monitor roughly 2.5 feet away. Since optics are used to create an image that looks large, despite the fact that the microdisplay module is small, these images are often called “virtual images”. And in this patent we will sometimes call them “apparent images”. Preferred embodiments of this invention would normally be implemented on devices using microdisplays that can display color images at least 800 pixels wide and at least 600 pixels tall.
An important common characteristic of the preferred embodiments of the present invention is that they are designed to facilitate a seamless, natural transition between near-to-eye viewing of content on a device's microdisplay (or microdisplays) and arms'-length viewing of content on a direct-view display. Key to enabling this seamless transition is software, hardware and storage mechanisms that are used to intelligently coordinate images on the microdisplay with images on the direct-view display. Using the descriptions in this patent and the figures for this patent (which help illustrate some of the graphical user interface elements of the invention disclosed herein), ordinary hardware and firmware engineers skilled in the art of developing computing and communication devices will be able to implement the hardware, and storage inventions disclosed herein, and ordinary programmers skilled in the art of software engineering on hand-held devices will be able to write their own code to implement the software inventions disclosed herein for devices that embody the hardware and storage inventions disclosed herein.
A preferred embodiment of my invention is a “smart-phone” supporting functions such as instant messaging, email, and contact management, in addition to phone calling, with both a microdisplay and a direct-view display. Since most instant messages, for example, are relative short text messages, this smart-phone could display instant messages on the direct-view display so that users can see the messages at arm's length, and can concurrently see the keys, buttons and controls on the phone handset so the user can easily type replies to the messages. When an instant message includes a reference to a Web page URL, or includes an attached photo or other relatively large high-resolution image or document, then when the user chooses to view that Web page or attachment, the smart-phone can display the Web page or attachment in the microdisplay instead of the direct-view display. This allows the user to see the Web page or attachment, which may be relatively large and high-resolution, by bringing the microdisplay to the user's eye, much like a PC display like image.
Additionally, when the device is displaying an image (or is preparing to display an image) in one display but not the others, then the device can display a text or graphics indication on one or more of the displays that is not displaying that image, indicating that one of the other displays is displaying that image (or that the device is preparing to display that image). For example, if the user sees an instant message in the-direct-view display (on a device with one direct-view display and one microdisplay), and that instant message refers to a Web page URL, if the user chooses to view that Web page, then while the device is downloading or preparing the Web page for display on the microdisplay the device can display a flashing eye icon on the direct-view display; and when the Web page is ready for viewing on the microdisplay, then the device can display a static icon on the direct-view display. This lets the user know that the user can now bring the device near-to-eye to view the Web page in a high resolution, magnified form. The same technique can be used any time the user can operate functions using the direct-view display that result in content then appearing on the microdisplay, or vice-versa. (If the user has the device near-to-eye and operates a function that results in new content appearing on the direct-view display, the microdisplay can display some indication that there is new content to view on the direct-view display.) It is important to provide these cues, since a user typically can not see the content on a microdisplay when holding the device at arms-length, and a user typically can not see content on a direct-view display (or doesn't think to look at the direct-view display) when looking into a microdisplay that is near-to-eye.
Device designers can choose to use the direct-view display as the primary display for many smart-phone functions, including instant messaging, email, phone dialing, contact management, schedule management, calculator functions, and the like, and to use the microdisplay as a high resolution viewer coordinated with the direct-view display.
In the emerging age of Internet-enabled mobile devices, some of the most important embodiments of this invention involve coordinating microdisplays and direct-view displays in ways that allow people to comfortably access and interact with full-page Web content on pocket-size devices. One important way our invention does this is as follows: When a user is viewing a Web page (or other content) on a device's microdisplay held near-to-eye, the device should allow the user to position a cursor or a rectangular outline (or some other indication of a “region of interest”) on a particular part of the Web page, and then when the user moves the device out to arms'-length the user should be able to view that region of interest on the direct-view display—that is, view a subset of the larger image that appeared on the microdisplay. This patent will refer to this subset as a “region of interest subset” or a “region of interest”.
When the direct-view display displays this region of interest, the direct-view display acts as a window onto the larger image that the user had been viewing on the microdisplay. That is, the region of interest portion of the image seen on the microdisplay is mirrored on the direct-view display. The user should then be able to read text or interact with content in that region of interest, or use controls on the device to scroll the region of interest left, right, up, down, or other directions, to see different parts of the image other than the subset of the image currently shown in the direct-view display. If the user moves the region of interest while viewing the image in the direct-view display, then when the user returns to viewing the image in the microdisplay, the device can graphically indicate where the region of interest has moved—for example, by overlaying a rectangle in the shape of the region of interest (which will generally correspond to the pixel width and height of the direct-view display) on top of the image seen in the microdisplay. Device makers can choose not to explicitly show a rectangle or other explicit representation of the region of interest on the image viewed in the microdisplay, or choose to make this kind of explicit “region of interest” indicator optional for the user. Whether or not the region of interest is explicitly outlined on the image viewed in the microdisplay, the device can mirror the region of interest in the direct-view display, as described above. Also note that the region of interest does not necessarily have to be rectangular. For example, a device designer could choose to use a circular or other shaped direct-view display (for example, on a watch or a non-conventional cell-phone or PDA), and then choose to use a correspondingly shaped region of interest.
The usage model facilitated by this invention mirrors what happens when a person uses a desktop computer monitor: Typically a person briefly “takes in” the entire monitor display (i.e., looks at the entire image without focusing attention on any specific region of the display at first), and then focuses attention on a specific region of interest (such as text they want to read, or a text-insertion point where they want to type, or some other item of interest). This invention allows microdisplay-enhanced devices to model this behavior.
As a result, this invention enables better ergonomics and better power-consumption characteristics than previous designs for many types of microdisplay-enhanced devices. With this invention, a user can choose to briefly observe a large area of content displayed on the microdisplay, and quickly skim the large area of content to identify a region of interest where they want to focus attention—and then (if they want) they can bring the device to arms'-length to view and interact with that region of interest on the direct-view display. By allowing the user to view a large area of content on the microdisplay, users can quickly orient themselves and quickly skim the large area of content to find a specific region of interest—just as they do on desktop monitors. This is something users cannot do on the small direct-view displays available on pocket-sized phones, PDAs, and hand-held computers that do not include embedded or attached microdisplays. Then, by allowing users to move the region of interest on the direct-view display held at arms'-length, users can choose to do most of their reading, typing, and other interacting with content while holding the device at arms'-length (i.e. normal reading distance). Being able to frequently operate the device and interact with Web pages and other content while holding the device at arms'-length is more comfortable for many people than having to hold a device near-to-eye for long periods. And it is much easier to type text on a device held at arms'-length than a device held near-to-eye. In addition, when the user is not holding the device near-to-eye, the microdisplay can be turned off or idled, saving power—so devices that use this invention can use significantly less power than devices that leave microdisplays on continuously. Similarly, when the user is holding the device near-to-eye, direct-view displays on the device can be turned off or dimmed or idled, saving power.
Clearly this method of intelligently coordinating content on microdisplays with content on direct-view displays can be applied to a wide range of content in addition to Web pages—including content such as spreadsheets, presentations, word processor files, blueprints, video, database forms, and other types of images, documents and applications.
The primary motivitation for this invention has to make it feasible for people to comfortably access and interact with full size Web content (and other high-resolution or large visual content) on pocket-size devices with built-in or attachable microdisplays. But the present invention's embodiments related to intelligently coordinating direct-view displays and microdisplays are applicable even when the microdisplays are part of a wearable display connected to the device rather than being embedded in or attached directly to the device. Wearable displays can include one or two microdisplays. For example, Inviso Inc.'s binocular wearable display product concept called “eShades” included two microdisplays, one for each eye. If a user is wearing a wearable display and looking at the near-to-eye microdisplays of the wearable display, they may find it awkward to type, particularly if the user has to look at a device's keypad frequently to find the letter keys while typing. A monocular wearable display (i.e. one with just one microdisplay positioned in front of one eye) lets the user use the other eye to look at the keypad; and many binocular wearable displays (like Inviso Inc.'s eShades concept product) allow users to see below the displays and therefore see a device's keypad held below the field of view of the wearable displays. But it is awkward to have to continuously shift ones eyes between the keypad and the image presented by the microdisplay while typing. Most users would be more comfortable if the region surrounding the current text-entry point (which might be an editable text field on a Web-based form, for example) were mirrored in the direct-view display of the device the user is using to type, ideally with the direct-view display close to the keypad on the device. This allows the user to keep their eyes focused in a small area on the device while typing.